History preservation in a computer storage system

ABSTRACT

A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

TECHNICAL FIELD

[0001] The invention relates to storage systems for computers, andparticularly to systems designed for long-term storage of data.

BACKGROUND

[0002] In disk-based storage systems, there is usually a clearseparation between the primary storage function—which deals withproviding rapid and efficient access to active data—and secondarystorage mechanisms which deal with less active data, with long term dataprotection, and with maintaining archives of historical storagecontents.

[0003] These secondary functions have, for the most part, traditionallybeen handled using magnetic tape storage. Reasons for this include thefact that tape has been much cheaper than disk storage (and otheralternatives), and tape cartridges are easily transported to provideoffsite copies of data to protect against loss due to localizeddisasters.

[0004] For a number of years, the cost per byte of disk hardware hasbeen dropping at a much faster rate than that of tape hardware, makingdisk increasingly attractive as an alternative to tape as a medium forsecondary storage. Some of the properties of disk, such as low-latencyrandom access, clearly make it superior to tape as a secondary storagemedium. If, however, the superior properties of disk are exploited in asecondary storage system, then new challenges arise which did notpreviously exist with tape.

[0005] For example, since every hard disk drive includes the mechanismfor reading and writing the media that it contains, in a disk-basedsecondary storage system it becomes attractive to keep all data onlineat all times. This means that traditional mechanisms for protectingarchival data, based on physically isolating and protecting the storagemedia, become inapplicable. One could simply turn the disks intowrite-once media by disallowing deletions in hardware, but then deletionof old data that are no longer needed would also be prohibited.

[0006] Moreover, for low cost safe disk storage it may be attractive touse an object storage scheme, such as is described in Margolus et al.,“A Data Repository and Method for Promoting Network Storage of Data,” US2002/0038296 A1 (Mar. 28, 2002). An object storage system is like a filesystem without a built-in mechanism for organizing the files (“objects”)into a hierarchy. The clients of the object storage system must defineand implement any such mechanism, for example by storing directoryinformation in objects. This lack of built-in hierarchy separates out acomplicated issue from the implementation of the storage system itself.

[0007] In the example of Margolus et al. US 2002/0038296, security andprivacy considerations are addressed by assuming that the storage systemhas little or no access to information about the structure or nature ofthe data that it stores. This constraint adds an extra dimension to theproblem of safely allowing deletion of unnecessary data, whileprotecting necessary data from malicious or accidental deletion.

[0008] If deletion of unnecessary data is to be allowed, mechanisms areof course required for determining which data has become unnecessary.Traditional backup schemes maintain “snapshots” of storage systemcontents at predefined moments, discarding some snapshots as unnecessaryafter some period of time. File servers often use an on-disksnapshotting mechanism for short-term protection of files from datacorruption or accidental deletion. Commonly, this is implemented bysimply avoiding overwriting data that is needed for some existingsnapshot, and instead writing the new data to a new location (andmaintaining appropriate indexing information for finding the differentversions of files). A snapshot is created by declaring at some point intime that no data that exists at that point will be overwritten. Asnapshot is discarded by freeing storage resources that are not neededby any other snapshot, and are not currently in use.

[0009] Thus one definition of unnecessary data is data that is onlyneeded by discarded historical snapshots. The challenge of deleting onlyunnecessary data then requires reconciling this definition with theconstraints and structure of a distributed, private and secure storagesystem. For example, it may not be possible, in general, for a storageserver to determine which stored data is part of a given historicalversion, or even which historical versions exist. This problem iscompounded if some pieces of data are shared: different historicalversions of the same object, or even different objects, may all sharecommon pieces of data, for storage efficiency. These pieces may only bedeleted when they are no longer needed by any version of any object.Finally, there may be more sophisticated needs for the protection ofhistorical information than are provided by simple snapshotting.

SUMMARY

[0010] In one aspect, the invention features a method by which adisk-based distributed data storage system is organized for protectinghistorical records of stored data entities. The method comprisesrecording distinct states of an entity, corresponding to differentmoments of time, as separate entity versions coexisting within thedistributed data storage system, and assigning expiration times to theentity versions independently within each of a plurality of storagesites according to a shared set of rules, before which times deletion isprohibited.

[0011] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. The shared set of rules mayrequire that unexpired entity versions not be changed. The storagesystem may be adapted for storing an unstructured-set of entities. Theunstructured set may comprise more than a million entities, or more thana billion entities. The storage system may associate an entity with anidentifier chosen by the storage client. The storage system mayassociate an entity version with an identifier that depends on a hash ofits contents. A client of the distributed storage system may definemechanisms to organize the storage system into a hierarchical filesystem, with separately accessible entities playing the roles of filesand directories. Expiration times of entity versions may be extended,and extension periods for different versions may be specifiedindependently. An expiration time may be extended at the request of aclient of the storage system. Information about the entity may bereplicated to a plurality of storage sites, with the set of sites chosenbased on a hash. Entity versions may be accessed separately, withoutneeding to access a larger aggregate first. The plurality of storagesites may be located in different cities. No single individual may beallowed physical access to all of the plurality of storage sites.Administrative mechanisms may exist for overriding the deletionprohibition. No single individual may be given the authority to overridethe deletion prohibition at all of the plurality of storage sites. Theversions of the entity may be assigned deposit times, and the versionwith the latest deposit time may be considered current. Non-currentversions may be assigned expiration times. The deposit time may bespecified by a client of the distributed storage system. The deposittime may be based on the time the deposit reaches a storage site. Thedeposit time may be constrained to agree with the actual time that thedeposit reaches a storage site, to within predetermined limits. Theactual time may be determined by clocks at the storage site, operatingwithout reference to an external time standard. The actual time may bedetermined by clocks at the storage site, with a limit to a totalcorrection applied per fixed period using an external time standard. Noconstraint may be imposed if the deposit time specified by the client isearlier than the latest deposit time of any existing version of theentity. The entity may be used to record the history of a file in asource file system, and an historical version of the file may be addedfrom a separate record of the file system's history with a deposit timethat precedes the most current version of the entity. The imposition ofthe constraint may begin at a predefined event, before which eventversions of the entity may be deposited with deposit times that violatethe constraint. The predefined event may be the deposit of a version ofthe entity with a deposit time specified that agrees with the actualtime, to within predetermined limits. The predefined event may be arequest from a storage client to begin monitoring deposit times for theentity. A client of the distributed storage system may deposit recordsof a source file system's history into the storage system, with entitiescorresponding to files and directories, and the deposit times specifiedfor versions of entities may correspond to times associated with therecords. Two distinct entities, each of which holds records of thecontent of a file in the source file system during different timeintervals, may be linked within a third entity. The third entity may beassociated with a directory in the source file system. The expirationtime assigned to a non-current version may depend on when it wassuperseded as the current version. The expiration time assigned to thenon-current version may depend on the deposit time that was assigned toit when it was current. The expiration time assigned to the non-currentversion may depend on the deposit time assigned to the version thatsuperseded it as the current version. The expiration time assigned tothe non-current version may depend on the actual time when it wassuperseded as the current version. The storage client may supplyinformation that allows the storage system to associate a version withthe version that it supersedes as the current version. The informationsupplied by the storage client may allow the storage system to order theversions of the entity by deposit time. The expiration time may dependon the length of the time interval during which the version was current.The expiration time may depend upon which defined snapshot moments theversion was current during. The expiration time may depend on thedeposit times of non-current versions of the entity. A version may bedeposited, and the expiration time for it may be set by the storageclient. A version may be deposited, and a time interval during which itis presumed to have been current may be assigned by the storage client.The expiration time may depend on the time interval during which aversion is presumed to have been current. A plurality of versions of afirst entity which are deposited during a time interval may all havetheir expiration times extended to at least a first expiration time. Asecond entity which records hierarchical directory information includingthat of the first entity may have a version deposited during the timeinterval which expires earlier than the first expiration time. Summaryinformation may be stored in a version of the second entity that doesnot expire before the first expiration time, that is sufficient torecreate hierarchical directory information of the version that does. Aversion may make reference to constituent blocks of stored content, witheach block assigned a reference count which reflects the number ofreferences there are to the block in any version. The version may bedeleted by a storage client, the reference counts assigned to itsconstituent blocks of stored content may be decremented, and a blockwith reference count of zero may be discarded and its storage space maybe reused. Versions may make reference to constituent blocks of storedcontent, with each block assigned a reference count which reflects thenumber of references there are to the block in current versions. Eachblock may also be assigned an expiration time that depends on the latestof expiration times associated with versions which make reference to it.A block which has a reference count of zero and an expiration time whichhas passed may be discarded, and its storage space may be reused. Thereference counts for blocks of stored content may be incremented whenthe blocks are deposited. The expiration time for a block of storedcontent may be set to a default non-zero value when the block isdeposited. Entities may be associated with entity version records, witheach entity version record storing the association between an entityidentifier freely chosen by a storage client and the versions of theentity. Each entity version record may be assigned a reference countwhich reflects the number of references there are to the correspondingentity from within current entity versions. Each entity version recordmay also be assigned an expiration time that depends on the latest ofall of the expiration times associated with the versions of the entityrecorded in the version record. An entity version record with referencecount of zero and an expiration time which has passed may be discardedand the storage space may be reused. The expiration time for an entityversion record may be set to a default non-zero value when it iscreated. The blocks of stored content may be strings of bytes with apredetermined maximum length. A block may be referenced using a blockname which depends upon a hash of the content of the block. The blockcontent may be encrypted using a key derived from its unencryptedcontent.

[0012] In another aspect, the invention features a method by which adisk-based distributed data storage system is organized for protectinghistorical records of stored data entities. The method comprisesrecording distinct states of an entity, corresponding to differentmoments of time, as separate entity versions coexisting within thedistributed data storage system, associating time-intervals with entityversions, corresponding to the times during which each entity versionwas considered current, sharing a set of rules for retaining entityversions among a plurality of storage sites, and designating some entityversions as deletable and some as undeletable independently at each ofthe plurality of storage sites.

[0013] Preferred implementations of this aspect may incorporate one ormore of the following. Except for deletion, entity versions may beimmutable. Expiration times may also be assigned to some entityversions, independently within each of the plurality of storage sites,according to a shared set of rules, before which times deletion may beprohibited. No single individual may be given the authority to overridethe deletion prohibition at all of the plurality of storage sites.

[0014] In another aspect, the invention features a method by which adisk-based data storage system is organized for protecting historicalrecords of stored data entities. The method comprises recording distinctstates of an entity, corresponding to different moments of time, asseparate entity versions coexisting within the data storage system,assigning expiration times to the entity versions, before which timesdeletion is prohibited, and assigning expiration times to blocks ofstored content that constitute the entity versions, with at least oneblock shared between different entities.

[0015] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. The data storage system may bedistributed and the expiration times may be assigned independentlywithin each of a plurality of storage sites according to a shared set ofrules. The expiration time assigned to a block may reflect the latest ofthe expiration times associated with a plurality of versions which makereference to it. A block may be assigned a reference count whichreflects the number of references there are to the block in a pluralityof versions which are not scheduled to expire. A block may be assigned areference count which reflects the number of references there are to theblock in a plurality of versions which are scheduled to expire duringsome specified finite time period. The block may also be assigned adefault expiration time that depends on a time of origin associated withthe block itself. The default expiration time may depend upon theexpiration times assigned to each of a plurality of versions which makereference to the block. A block with a reference count of zero and adefault expiration time which has passed may be discarded and itsstorage space may be reused. An authorized storage client may cause ablock to be discarded which has a default expiration time which has notyet passed. An authorized storage client may cause a version to bedeleted for which the assigned expiration time has not yet passed. Ablock referenced by the deleted version may be discarded and its storagespace may be reused.

[0016] In another aspect, the invention features a method for keepingtrack of when all references of a specified category made to elementshave been removed. The method is designed to fail in a manner that doesnot falsely conclude there are no references. The method comprisescomputing a hash value that identifies the source of a reference,combining hash values using a first operation to record the addition ofreferences, combining hash values using a second operation to record theremoval of references, and concluding that reference additions for anelement have been matched by reference removals.

[0017] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. The hash may be acryptographic hash. The first operation may include counting the numberof reference additions. The first operation may include adding togetherhashes. The first operation may include adding corresponding bits ofhashes together modulo 2. The hash value uniquely may identify thereference source. Additional information not needed to identify thereference source may be included in the identifying hash. Hash valuesmay be combined at a physical location that is separated from a sourceof references. The additional information may be examined at thelocation where the hash values are combined, and a decision may be madeto not combine a hash value. The additional information may be examinedat the location where the hash values are combined, and determines whichcategories of combined hash will be affected. A reference-removaloperation may be performed on one category of combined hash and areference-addition operation may be performed on another. Referencesources and combined hashes may be distributed among a collection ofcomputers. The computers may be servers in a disk-based data storagesystem. The data storage system may be organized for protectinghistorical records of stored data entities. Distinct states of an entitymay be recorded, corresponding to different moments of time, as separateentity versions coexisting within the data storage system. Expirationtimes may be assigned to the entity versions, before which timesdeletion is prohibited. Expiration times may be assigned according to ashared set of rules. Expiration times may be assigned to blocks ofstored content that constitute the entity versions. A hash value mayidentify the reference of an entity version to a block that is sharedwith other entities. Information about the shared set of rules may beincluded in the reference-identifying hash. Information that allows thegeneral deletion prohibition to be ignored may be included in thereference-identifying hash. Reference additions to the shared block mayhave been matched by reference removals, and the shared block may bediscarded and its storage space may be reused.

[0018] In another aspect, the invention features a method by which morethan one client program connected to a network stores the same data itemon a storage device of a data repository connected to the network. Themethod comprises encrypting the data item using a key derived from thecontent of the data item, determining a digital fingerprint of the dataitem, storing the data item on the storage device at a location orlocations associated with the digital fingerprint, and assigning anexpiration time to the data item, before which time deletion isprohibited.

[0019] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the data item may depend upon expirationtimes assigned by the client programs.

[0020] In another aspect, the invention features a method by which morethan one client program connected to a network stores the same data itemon a storage device of a data repository connected to the network. Themethod comprises determining a digital fingerprint of the data item,testing for whether the data item is already stored in the repository bycomparing the digital fingerprint of the data item to the digitalfingerprints of data items already in storage in the repository,challenging a client that is attempting to deposit a data item alreadystored in the repository, to ascertain that the client has the full dataitem, and assigning an expiration time to the data item, before whichtime deletion is prohibited.

[0021] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the data item may depend upon expirationtimes assigned by the client programs.

[0022] In another aspect, the invention features a method by which morethan one client program connected to a network stores the same data itemon a storage device of a data repository connected to the network. Themethod comprises determining a digital fingerprint of the data item,storing the data item on the storage device at a location or locationsassociated with the digital fingerprint, associating the data item witheach of a plurality of access-authorization credentials, each of whichis uniquely associated with an access owner, assigning an expirationtime to the data item, before which time deletion is prohibited, andpreparing a digital time stamp of a plurality of records associatingdata-items and credentials, to allow a property of these records to beproven at a later date.

[0023] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the data item may depend upon expirationtimes assigned by the client programs.

[0024] In another aspect, the invention features a method by which morethan one client connected to a network stores the same data item on astorage device of a data repository connected to the network. The methodcomprising determining a digital fingerprint of the data item, testingfor whether a data item is already stored in the repository by comparingthe digital fingerprint of the data item to the digital fingerprints ofdata items already in storage in the repository, associating with a dataitem an informational tag which may be read by at least some clientprograms, and assigning an expiration time to the tagged data item,before which time deletion is prohibited.

[0025] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the tagged data item may depend uponexpiration times assigned by the client programs.

[0026] In another aspect, the invention features a method by which aclient connected to a data repository over a lower speed networkconnection may provide higher speed access to a data item forapplication processing than is possible over the relatively low speedconnection to the network, the method comprising determining a digitalfingerprint of the data item, testing for whether the data item isalready stored in a repository by comparing the digital fingerprint ofthe data item to digital fingerprints of data items already in therepository, only if the data item is not already in the repository,transferring the data item over the lower speed connection from theclient to the repository, assigning an expiration time to the data item,before which time deletion is prohibited, making a higher speedconnection between an application server and the data repository,executing an application on the application server to process the dataitem stored on the data repository, and returning at least some of theprocessed data to the client across the lower speed connection.

[0027] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the data item may depend upon expirationtimes assigned by the client programs.

[0028] In another aspect, the invention features a method by whichmultiple clients browse content on a network such as the Internet. Themethod comprises each of the multiple clients accessing content on thenetwork via one or more proxy servers, determining the digitalfingerprint of an item of content passing through the proxy server,storing the item of content in a content repository connected to theproxy server at a location associated with the digital fingerprint,assigning an expiration time to the item of content, before which timedeletion is prohibited, testing for whether a content data item isalready stored in the repository by comparing the digital fingerprint ofthe content data item to the digital fingerprints of content data itemsalready in storage in the repository, and associating a content dataitem already stored in the repository with an access authorizationcredential uniquely associated with an access owner.

[0029] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the item of content may depend uponexpiration times assigned by the multiple clients.

[0030] In another aspect, the invention features a method by whichclients store content items which are broken into up into smaller dataitems in a data repository connected to the network. The methodcomprises determining a digital fingerprint of a data item, testing forwhether a data item is already stored in the repository by comparing thedigital fingerprint of the data item to the digital fingerprints of dataitems already in storage in the repository, and assigning an expirationtime to a data item, before which time deletion is prohibited.

[0031] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. Rules governing expiration anddeletion may be distributed among a plurality of storage sites. Theexpiration time assigned to the data item may depend upon expirationtimes assigned by the multiple clients. The expiration times assigned todata items that comprise a content item may depend upon an expirationtime assigned to the content item. The content item may be broken up ina manner that is independent of the content. The content item may bebroken up in a manner that depends on the content type. The content itemmay be broken up at boundaries defined by predetermined byte strings.The choice of which byte strings constitute boundaries may depend uponthe value of a hash function acting on the byte strings.

[0032] In another aspect, the invention features a method for ensuringthat rules that prevent premature deletion of entity versions areenforced by correctly operating servers that store the blocks of contentthat comprise the entity versions. The method comprises computing a hashvalue that identifies the source of a reference to a block of content,incorporating into the hash value a description of rules or parametersthat are needed in order to enforce rules, and communicating informationwhich allows the hash value to be computed, to a server that stores theblock of content.

[0033] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. The hash may be acryptographic hash. The hash value may uniquely identify the referencesource. A block of content may be identified by a digital fingerprintthat involves a hash of its content. A block of content may be assignedan expiration time, before which time deletion may be prohibited. Theblocks of content may be distributed among a plurality of storage sites.An expiration time assigned to an entity version may also be assigned toeach of its constituent blocks of content. The information which allowsthe hash value to be computed may be included in a request to delete theblock of stored content. A server storing the block of content may denya request that violates a rule or parameter specified in the informationsupplied when the block was created. Distinct states of an entity may berecorded, corresponding to different moments of time, as separate entityversions coexisting within a data storage system. The rules governingdeletion of an entity version may depend upon when the entity versionwas created. Hash values that identify references to blocks of storedcontent may be combined as part of a reference counting scheme. Somereference counts may be associated with expiration times, and theirvalues may be ignored after some point in time. The connection betweenan entity version and a constituent block of content may not be visibleto a server storing the block of content. The stored block of contentmay expire and the server storing it may discard it and may reuse itsstorage space. The information supplied by the storage client thatassociates a version with a superseded version may be discarded whilethe two versions are retained.

[0034] In another aspect, the invention features a method by which adistributed disk-based data storage system is organized for protectinghistorical records of stored data entities. The method comprisesrecording distinct states of an entity, corresponding to differentmoments of time, as separate entity versions coexisting within the datastorage system, assigning expiration times to the entity versions,before which times deletion is prohibited, assigning expiration times toblocks of stored content that constitute the entity versions; andassigning a reference count to a block of stored content that reflectsthe number of references there are to the block in entity versions whichare scheduled to expire during some specified finite time period.

[0035] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. The block may also be assigneda reference count that reflects the number of references there are tothe block which are not scheduled to expire. The block may also beassigned a default expiration time which sets an earliest time that theblock can expire, even if all expiration related reference counts arezero. The data storage system may be distributed and the expirationtimes may be assigned independently within each of a plurality ofstorage sites according to a shared set of rules. An authorized storageclient may cause a block to be discarded and its space may be reusedwhen its expiration time has not yet passed. An authorized storageclient may override the deletion prohibition and may cause an entityversion to be deleted when its expiration time has not yet passed. Ablock of stored content referenced by the deleted version may bediscarded and its storage space may be reused.

[0036] In another aspect, the invention features a method by which adisk-based data storage system is organized for protecting historicalrecords of stored data entities. The method comprises recording distinctstates of an entity, corresponding to different moments of time, asseparate entity versions coexisting within the data storage system, andassigning finite expiration times to entity versions based oninformation supplied by the storage client, before which times deletionis prohibited and after which times deletion is allowed.

[0037] Preferred implementations of this aspect of the invention mayincorporate one or more of the following. A version may be deposited,and the expiration time for it may be set by the storage client. Aversion may be deposited, and a time interval during which it ispresumed to have been current may be assigned by the storage client. Theexpiration time may be assigned by a storage server and may depend onthe time interval during which a version is presumed to have beencurrent. The entity may be used to record the history of a file in asource file system, and an historical version of the file may be addedfrom a separate record of the file system's history. The addedhistorical version may have an interval during which it is presumed tobe current specified that predates that of an existing version of theentity. Expiration times of entity versions may be extended, andextension periods for different versions may be specified independently.Unexpired entity versions may not be changed. The storage system may beadapted for storing an unstructured-set of entities. The connectionbetween an entity version and a constituent block of content may not bevisible to a server storing the block of content. A plurality ofversions of a first entity which are deposited during a time intervalmay all have their expiration times extended to at least a firstexpiration time. A second entity which records hierarchical directoryinformation including that of the first entity may have a versiondeposited during the time interval which expires earlier than the firstexpiration time. Summary information may be stored in a version of thesecond entity that does not expire before the first expiration time,that is sufficient to recreate hierarchical directory information of theversion that does. Versions may make reference to constituent blocks ofstored content, with each block assigned a reference count. Each blockmay also be assigned an expiration time that depends on the latest ofexpiration times associated with versions which make reference to it. Ablock which has a reference count of zero and an expiration time whichhas passed may be discarded, and its storage space may be reused.

[0038] Other features and advantages of the invention will be apparentfrom the drawings, detailed description, and claims.

DESCRIPTION OF DRAWINGS

[0039]FIG. 1 is a block diagram showing a storage clique (storage site)consisting of four storage servers, connected to some backup clientsover a local area network.

[0040]FIG. 2 is a block diagram showing three storage sites of an objectstorage system, one in Atlanta, one in Boston, and one in Cairo. Each ofthe sites has independently assigned expiration times to three versionsof object f.

[0041]FIG. 3 is a block diagram showing four objects (a, b, c and d)with new versions deposited at the indicated times. Three snapshotmoments are defined, at times t1, t2 and t3.

[0042]FIG. 4 shows an example history of expiration times for adatablock. The block is first deposited on day 47 of 2003, itsexpiration time changes as versions referencing it are deposited anddeleted, and finally the block expires and is discarded on day 147 of2003.

[0043]FIG. 5 shows an example of additional information that might becommunicated when object versions that references the datablock of FIG.4 are added or removed. The reference counts in FIG. 4 are replaced withsums of hash values that depend on this additional information.

[0044]FIG. 6 is a block diagram showing three storage sites of an objectstorage system, one in Atlanta, one in Boston, and one in Cairo. Each ofthe sites has independently marked the least stable version of d fordeletion.

DETAILED DESCRIPTION

[0045] There are a great many different implementations and embodimentsof the invention possible, too many to possibly describe herein. Somepossible implementations that are presently preferred are describedbelow. It cannot be emphasized too strongly, however, that these aredescriptions of implementations of the invention, and not descriptionsof the invention, which is not limited to the detailed implementationsdescribed in this section but is described in broader terms in theclaims.

Glossary of Terms

[0046] The definitions below reflect the usage of some of the terms usedin this document.

[0047] Clique: A cluster of servers at a single storage site.

[0048] Coalescence: The merging of datablocks with the same dataname.

[0049] Collision: The accidental agreement of hashes of distinct data.

[0050] Content Hash: Cryptographic hash of the contents of a datablock.

[0051] Continuous History: A complete record of current and historicalversions.

[0052] Cryptographic Hash: A hash designed to never have collisions.

[0053] Currency Interval: The interval during which a version wascurrent.

[0054] Current Version: The object version with the latest deposit time.

[0055] Datablock: A string of bytes of bounded size.

[0056] Dataname: Content hash of a datablock, used as its uniqueidentifier.

[0057] Deposit: The process of sending data to the object storagesystem.

[0058] Directory Object: An object used to record file hierarchyinformation.

[0059] Directory Version: A version of a directory object, current orhistorical.

[0060] Expiration Time: A time after which deletion is allowed.

[0061] Gateway: An application server acting as a storage system client.

[0062] Hash: A psuedo-random map from a byte string to a fixed-lengthvalue.

[0063] Historical Version: An object version that is not the currentversion.

[0064] Metablock: Storage server representation of an object,specifically a record linking an object identifier to a version list.

[0065] Namespace: A related set of object identifiers.

[0066] Object: A collection of object versions named by an objectidentifier.

[0067] Object Identifier: A client-specified identifier for an object.

[0068] Object Storage System: A system for storing unstructured sets ofobjects.

[0069] Object Version: A set of byte strings, e.g., a set of datablocks.

[0070] Policy: A version retention policy.

[0071] Reference Count: A count of the number of references to anentity.

[0072] Replica: A redundant copy of a block of data, used for dataprotection.

[0073] Scalable: Able to be increased in size and capacity indefinitely.

[0074] Self-Backing: Storage that does not require external backup.

[0075] Self-Encrypted: Encrypted in a key that depends on theunencrypted data.

[0076] Self-Named Data: Data named by a content hash of the data.

[0077] Server: A storage server.

[0078] Snapshot: A set of object versions that were current at aspecified moment.

[0079] Snapshot Policy: A policy that assigns expiration times tosnapshots.

[0080] Stable Version: A version which remains current for a relativelylong time.

[0081] Stable Version Policy: A policy that retains stable versions.

[0082] Storage Server: A dedicated computer that provides storageservices.

[0083] Storage Site: A place where a portion of the storage system islocated.

[0084] Version: An object version.

[0085] Version List: A list of versions that make up an object.

[0086] Version Retention Policy: A shared set of rules used by storageservers to govern when versions can be deleted.

[0087] Version Thinning: The process of deleting versions that haveexpired.

Introduction

[0088]FIG. 1 shows a system diagram of one storage site (storage clique)consisting of several storage servers of an object storage system. FIG.2 shows several such sites connected together over a wide area network,linked together as parts of a geographically distributed storage system.FIG. 1 also shows a selection of storage clients, each of whichcommunicate with the storage clique using a defined object storagesystem protocol.

[0089] Client A is a file server running a backup program, which allowschanged file and directory information on the file server to be storedas new versions of objects in the storage system, with each file anddirectory corresponding to a distinct object, and each object havingmultiple versions, corresponding to distinct historical states of theobjects. Client B is an NFS gateway, which presents the file systeminformation stored by client A (including historical information) as aread-only file system using the NFS file sharing protocols. Client C isa tape server, which talks to a tape drive.

[0090] The example storage system of FIGS. 1 and 2 is an elaboration ofthe one described in Margolus et al., US 2002/0038296 A1. The overallstorage system is comprised of a collection of geographically separatedstorage centers called “cliques,” each of which is comprised of one ormore “storage servers,” which in turn are comprised of one or moreprocessors and storage devices. Some of the desirable properties of thepreferred embodiment of the object storage system are:

[0091] (1) Fault Tolerance. The system is composed of a number ofservers and a number of cliques, and there are no single points ofhardware failure. Data is stored redundantly. If some predeterminedmaximum number of servers are removed from a clique, no data is lostwithin that clique. If a predetermined number of cliques are removedfrom the system, no data that has had time to be communicated to othercliques is lost. Geographic separation makes it unlikely that a disasteraffecting one clique will damage another.

[0092] (2) Hash-Based Data Names. Datablocks (called data items inMargolus et al. US 2002/0038296 A1) are strings of bytes with apredetermined maximum size that are the basic unit of content. Acryptographic hash of the contents of a datablock, called a dataname, isused to name the datablock in the system, to locate the datablock withinthe system, and to test whether a new datablock being deposited alreadyexists within the system, and hence can be communicated and stored byreference, without transmitting or storing the actual bytes of thedatablock.

[0093] (3) Self Encryption. Datablocks may be stored in an encryptedform. If the encryption key is derived from the unencrypted contents ofthe datablock, then different storage clients will independently turnthe same unencrypted block into the same encrypted block, and sosecurity can be achieved while still avoiding the need to transmit andstore the same information repeatedly.

[0094] (4) Objects and Versions. The storage system associates an objectidentifier, chosen by a storage client, with stored data. Each objectcan include many object versions of the stored data, each of which isidentified by its deposit time. An object version references some set ofconstituent datablocks using their datanames. Many object versions mayreference the same datablock.

[0095] In addition to protecting data against server faults and cliquedestruction, the storage system is designed to allow recovery fromaccidental or malicious deletion or corruption. The primary mechanismfor achieving this is based on the notion of object history. Storageclients define and name data objects which change their contents withtime, but not their names. Each named object may include multipleversions, the latest of which is the “current version;” the rest are“historical versions.” In case a client deletes or corrupts the currentversion, the preservation of some number of historical versions allowsrecovery. Historical versions may also have archival value.

Version Retention

[0096] In a distributed object storage system, robustness againstphysical damage to individual cliques comes from redundancy acrosswell-separated cliques. This same property of geographic separation,together with abstraction barriers that isolate the internal operationof cliques from outside control, provides an effective avenue forprotecting object history. A shared set of rules governing which objectversions must be retained, and for how long, are enforced independentlywithin each clique: this independent enforcement of a version retentionpolicy makes it impossible for any accidental or malicious act that ismediated by one clique to cause any other correctly operating clique todelete any data that is not determined by the policy to be deletable.Care must also be taken that ordinary storage system operators andadministrators have no special privileges or physical access that letthem globally circumvent or change these policies.

[0097] Useful version retention policies can be based on how long aversion remains current. At the moment when a new version is written,the time interval during which the superseded version was currentbecomes apparent: the version was current from the time it was createduntil the moment it was superseded (or deleted). For this time intervalinformation to be visible to the storage servers, it must be the casethat a new version is created by referencing the current version. Thiswould not be the case, for example, in a storage system in which newversions are completely independent objects as far as the storageservers are concerned, and their linkage to existing objects is onlyvisible to the storage client.

[0098] Of course, the connection between different object versions doesnot have to be continuously apparent to the storage system: the linkagebetween apparently independent objects in the storage system could beindicated by the storage client at the moment when a new version isbeing written. At that moment, an object version marked “current” couldbe changed to “historical” (and an expiration assigned), while a newobject version marked “current” could be created.

[0099] Since the storage client needs a mechanism for accessing objecthistory in order to be able to “roll back the clock” to a an earlierstate, and since aggregating information that associates object versionsreduces the number of entities that the storage system must manage, itmakes sense for the system to have a form of “object metadata” thatlinks the various object versions together, and to the objectidentifier. This is called a metablock, and it provides access to a listof versions that includes dataname references to the actual datablocksthat constitute the versions. If the version-list provides the only linkbetween versions, then if the version list is encrypted between accessesusing key information provided by the storage client at access time, andthis key information is not stored, then the linkage between versions isnot normally visible to storage servers. A metablock is only one exampleof how an object with multiple versions may be represented in a computersystem.

[0100] At the moment a version is superseded, an expiration time can beassigned to it based on the time interval during which it was current(and perhaps other factors). Historical versions are then kept safe bythe server policy of not allowing unexpired versions to be deleted. Aslong as servers constrain historical-version expiration based on thetime interval during which the version was current, storage clientscannot subvert this mechanism. A malicious system administrator withwide sweeping file access privileges could conceivably delete all of thecurrent files to which he has access, but this would define the end ofthe current time-interval of all of those formerly current versions, andthey would be assigned expiration times by the servers which he couldnot hasten. Earlier historical versions that had not expired would alsobe protected. Even if a malicious agent was able to gain total controlof a storage clique at one location, it would have no power to deletethe redundant information about unexpired versions stored at otherlocations, or hasten their expiration.

[0101] Expiration assignment is illustrated in FIG. 2. Three cliques areshown in three different cities. Some of the version list information oran object f that is represented at each of these three storage sites isshown in the illustration. For each of three versions, f1, f2 and f3,each of the three cliques independently computes expiration times andstores them in the version list, during a period when the version listis accessible. Each clique independently prohibits deletion of unexpiredversions.

[0102] Note that, as long as the expiration time assigned to a versiondepends on the time interval during which the version was current, themoment when the version is superseded must be determined (or at leastmonitored) by the clique. If this time was specified solely by thestorage client, then a malicious client could subvert the protectionmechanism by specifying old deposit times for new versions, thus makinglong-lived current versions appear ephemeral (and hence they would notbe protected). If the replacement time is determined independently bythe clique, or at least monitored by the clique to be in reasonableagreement with actual time, then this problem does not arise.

[0103] This mechanism requires a dependable time standard. There isclearly a danger that any external time standard used by a clique mightbe compromised, resulting in incorrect and unsafe behavior. One way todeal with this is to have each clique act as its own time standard: onceits clocks are set during system startup (or restart), they comparethemselves only with each other. It would also be relatively safe to letthe clique use an external time standard as long as the clique refusesto change its clocks by very much in the course of a day. This preventsa malicious agent from doing much damage by manipulating the timestandard.

[0104] Note also that minimum guarantees of persistence of history arenot endangered by allowing storage clients to delay the expiration ofhistorical versions of objects, or make them permanent.

Snapshot Retention

[0105] One useful version retention policy is based on the idea of filesystem snapshots discussed earlier. An understandable and usefulretention policy is to simply guarantee access to a complete snapshot ofall objects belonging to a given storage client at specified snapshotmoments, with each snapshot guaranteed to persist for a specified periodof time.

[0106] This kind of policy is similar to conventional practice with theretention of backup tapes. For example, if the snapshot moments arechosen to occur every day at some fixed time, then these snapshotscorrespond to daily full tape backups. If some daily snapshots are keptonly for a week, while others are kept for a month or a year or madepermanent, then this policy provides the same level of recoverabilityfrom client deletion or corruption of current data as conventional tapebackup.

[0107] Retaining snapshots does not involve actively making copies ofobject versions. It is only as a new object version is written that itmust be determined how long to keep the previous version, in order toretain the information needed for reconstructing historical snapshots.For each current version, as it is superseded, it is apparent exactlywhich “daily backups” its time interval of currency covered, and if eachof these backups has an expiration associated with it, then theexpiration assigned to the version at this moment is simply the latestof all of these.

[0108] The relationship of versions and snapshots is illustrated in FIG.3. Three snapshot moments are defined, at t1, t2 and t3, and fourobjects are shown, a, b, c and d. We'll assume for the sake ofillustration that to is the beginning of the operation of the objectstore, and nothing changes after the times illustrated. Object a iscreated and then does not change until after t3, and so version a1remains current for all three snapshots. For object b, version b1 iscurrent for snapshot 1, and version b2 is current thereafter. For objectd, only three of its versions (d3, d5 and d7) are current at snapshotmoments. If we're following a snapshotting version retention policy,then the rest of the versions of d can be deleted as soon as they aresuperseded. If snapshot 1 items get an expiration of a day, snapshot 2 aweek, and snapshot 3 a month, then a1 lasts a month (measured from t3),c1 lasts a week (starting from t2) and b1 lasts a day (starting fromt1). If nothing changes after the times illustrated, then versions a2,b2, c2 and d7 will remain current indefinitely, and so are not subjectto expiration.

[0109] Notice that a snapshot has both a time at which it is taken andan expiration period. E.g., a snapshot may be taken every day, but somehave expiration periods of a day, others a week, and still others amonth.

[0110] Note that if the object storage system is being used forprotecting a record of data that is actively used on some other“primary” storage system (i.e., backup), then as long as changes to theprimary storage system are eventually copied into the object store, eachchanged file version (for example) will appear as the current objectverson during some set of snapshot moments, and will be protectedaccordingly.

Adding History

[0111] Traditional full backup of disk storage results in snapshots ontape media. These have the problem that they have a finite lifetime, notonly because the tapes degrade with time, but also because the mediabecome obsolete and it becomes difficult to find or maintain hardwarethat can read old tapes. Thus in cases where tape backup is beingdisplaced by versioned object storage, it may be desirable to be able totransfer old backup-tape snapshots into the object storage system, toallow the tape media to be retired (see client C in FIG. 1). This may bedone long after an object storage system has taken over the role ofdirectly backing up a primary storage system (e.g., client A in FIG. 1).

[0112] One way to add history to an existing object storage system is tohave storage clients directly write historical versions of existingobjects. The current time-intervals associated with these versions areknown at deposit time and so expiration times can be either assigned tothem by the storage system or specified explicitly by the storage clientas they are written. It is natural, for example, to add full-backups ofa file system in this manner, and the backup snapshots can be added inany order. Allowing extra historical versions to be directly added doesnot affect the protection afforded by the expiration times assigned toexisting historical versions.

[0113] Another way to deal with this is to add the history using a setof objects distinct from any existing objects. This has the advantagethat incremental backup tapes recorded from a file system can be loadedin historical order to create a full snapshot from each incrementaltape, as each successive tape defines new current versions. If the datesassociated with the data on the tapes are assigned to the creation andreplacement times for versions, then the object storage system candetermine the current time-intervals associated with those versions byitself, and hence assign expiration times automatically, according toset policy.

[0114] Allowing the operation of “deposit with an old date” does notinterfere with the protection afforded object versions which are notbeing added in from historical records, as long as only current versionswhich were deposited with a significantly out-of-date deposit time canbe superseded with one. This rule prevents, for example, a currentversion which was deposited a year ago with an up-to-date deposit timefrom being replaced now with a current version with a deposit time ofone second later

[0115] If all incremental tape-history of a source file system is addedbefore any new history is added (i.e., before one begins to back up thecurrent state of the source file system directly into the object storagesystem), then the same objects can be used for both the tape-history andthe continuing backup. If, on the other hand, old history fromincremental tapes is to be deposited after new history has already begunto be accumulated by direct backup into the storage system, then the twosets of objects can be explicitly linked, to make all of the historyconveniently accessible to the storage clients. This can be done, forexample, by recording the association in appropriate root directoryobjects.

Continuous History

[0116] If, for some set of objects, no versions deposited during sometime period are deleted, then any moment of the history of that set ofobjects during that period can be reconstructed. The reconstructionconsists of the set of object versions that were current at the givenmoment during that period. For example, if a file system is stored inthe object storage system and no versions of files or directories aredeleted for the first hour that they exist, then any moment of historyfor the most recent hour can be reconstructed as a snapshot.

[0117] Both continuous history and discrete snapshots can be combined.For example, if the finest-grained discrete-snapshot interval is hourly,then object versions that change more often than that would normally beoverwritten, rather than have historical versions kept. If the storageclient simply extends the expiration time of all object versions to beat least one hour after the moment they are superseded as the currentversion, then all versions of all objects for the most recent hour willbe available. Thereafter, snapshots will be available according to thepredetermined version retention policy, unless expirations areexplicitly extended. If, as another example, all current object versionssuperseded during some particular hour are extended to a month, thencontinuous history will be available for that particular hour, for amonth.

[0118] If the storage client is using the object store directly as afile system and recording file system directory information in ordinaryobjects deposited in the object store, then there may be a very largenumber of directory versions retained to allow this kind of continuoushistory reconstruction. This can be greatly reduced by being selectiveabout which versions of directories have their expirations extended, andby including extra transaction-log information in the retained directoryversions that allows the deleted versions to be reconstructed.

[0119] For example, the client might only keep directory versions whenclient-defined directory metadata changes (e.g., permissions of who isallowed to access the directory). Each retained version is associatedwith a start time and an end time for the period summarized by theversion; a record of the ending contents of the directory; and atransaction log, including the timing, of all file additions, deletions,file renamings and file movements between directories that have affectedthis directory during this time period. This allows any intermediateversion of the directory to be recreated, without keeping explicitversions.

[0120] The interaction of summary directories with a snapshotting policycan be illustrated with reference to FIG. 3. Suppose that the intervalfrom t0 to t3 is three hours, divided into three equal intervals in thepicture. Suppose also that continuous history is being kept for onehour, that object d is a directory, and that d's directory metadata doesnot change during the interval t0 to t3. This means that, except for theextra directory versions (d3 and d5) that are automatically retained atthe snapshot moments, and the current version (d7), all other directoryversions written during the period shown can be deleted as soon as theyare superseded. Each directory version contains a summary of the pasthour, up to that version, and in particular version d7 allows areconstruction of directory contents of versions d4 through d7, d5 canreconstruct d1 through d5, and d3 can reconstruct d1 through d3. Toreconstruct a moment midway between snapshots 1 and 2, the directorycontents of version d4 is reconstructed from the directory contents andtransaction log of the next available directory version (d5), and a1, b1and c1 are part of the reconstructed moment. The extension of theexpirations of the non-current versions a1, b1 and c1 to a minimum ofone hour can be requested by the client as they are superseded, withonly the expirations associated with the discrete snapshot moments setby the clique.

[0121] To reconstruct a discrete snapshot moment long after thecontinuous history has expired, no intermediate directory versions needto be reconstructed. Since no directory version will be deleted whichwas current at any discrete snapshot moment that is still beingretained, directories for snapshot moments will always be available.Only the ending contents, recorded in the directory version current atthe snapshot moment, is needed.

[0122] Eventually, some of the retained directory versions will expireand may be deleted. Because of the log-start and log-end timesassociated with the continuous history directory versions, it willalways be clear which intervals of directory history can still bereconstructed with the available information.

Manual Version Thinning

[0123] One scheme for protecting version history while still allowingunnecessary versions to be deleted (and the corresponding unneededstorage to be reclaimed) is to leave the protection up to the servers,and the deletion up to the clients.

[0124] In this scheme, each datablock has a reference count associatedwith it. The reference count reflects how many times the block appearsas part of any version, historical or current. Current versions areassigned an expiration time by the storage server when they aresuperseded, according to a predetermined retention policy. Expirationscan be extended by storage clients. Expired versions can be deleted by astorage system client, but no other versions are allowed to be deleted.Thus necessary versions are protected from deletion by thestorage-server-enforced “retain until expired” policy, but deletionbecomes the responsibility of the storage client.

[0125] When an expired version is deleted, the reference counts of allof its constituent datablocks are appropriately decremented by thestorage server. A datablock may be deleted from storage, and its spacereclaimed, if its reference count is zero. Note that all reference countdecrementing can be performed by the storage server at the moment whenthe client explicitly deletes an expired version, and so versionmetadata may be encrypted in between times, using a key provided by thestorage client that the storage server uses transiently and does notretain.

[0126] This manual scheme has the drawback that it is necessary for thestorage client to periodically access all of its objects in order todelete unneeded versions. It is also difficult to turn into an automaticserver-side scheme. For example, one could imagine having the storageservers delete any expired versions any time an object is accessed. Thiswould work fine for ordinary files, but not for directory objects, whichwould have to be marked as special and handled directly by storageclients—since the server can not see inside them. If a directory versionwere automatically deleted by a storage server, objects that were onlyreferenced by that version would become unreachable by the storageclient, which could never access them again, and so the storage serverwould never reclaim their space. Both of these problems are fixed by theautomatic scheme outlined in the next section.

Automatic Version and Datablock Deletion

[0127] As long as all storage servers in the object storage system agreeon the algorithm for determining the expiration time of a version whenit is superseded as current, then both the version and all of thedatablocks that comprise it can be assigned this expiration time. If theexpiration time of a version is extended, each datablock that itreferences can be assigned the extended expiration time. If a datablockis referenced by several versions, its expiration time becomes thelatest of all of the expiration times assigned to it as part of eachversion. It is thus guaranteed that a datablock will not expire untilall of the non-current versions that it is part of have also expired. Ifthe datablock is not part of any current versions, then when it expiresit is safe to delete it. Otherwise it must not be deleted. Thus it isimportant to know whether a datablock is part of any current versions.Therefore each datablock has associated with it both an expiration date,determined by the latest expiring version it is part of, and acurrent-reference count, determined by how many current versions it ispart of. A datablock may be deleted if it is not part of any currentversions (i.e., its current-reference count equals zero) and it is notpart of any unexpired historical version (i.e., its expiration time haspassed).

[0128] Using this scheme, space can be reclaimed automatically by thestorage servers as soon as versions expire, even if an object is neveraccessed again. Furthermore, it makes no difference whether the versionsaffected are directories or not. All file and directory versions currentat a snapshot moment are kept at least until that snapshot expires. Fileand directory versions that were not current during any unexpiredsnapshot are not needed, and datablocks referenced only by them can bedeleted. For example, in FIG. 3, if snapshot 1 has expired, then thedatablocks referenced only by b1 are no longer needed. Those referencedby b2 are still current (assuming nothing changes after time t3). Ifsome datablock is referenced by b1 and c1 and by no other version, itwill not expire until snapshot 2 expires.

[0129] If there are current versions of files or directories that arenot reachable from the root of the file system, then these files willnever be superseded and their datablocks will never be released. Thissituation will never arise, however, as long as the file system is keptconsistent and children are deleted before their parent directories are.

[0130] Note that, when datablocks are first deposited in the storagesystem and before they have become part of any object version, it may beconvenient to guarantee that they persist for some minimum period. Thusdatablocks may be deposited with a non-zero expiration time. This hasthe effect of extending the minimum persistence, but it does not hinderthe ultimate reclamation of storage space. Alternatively, thecurrent-reference count may be incremented when the datablock isdeposited (or redeposited, in the case of shared datablocks). Thisavoids the need for a burst of reference-count increments when theversion referencing the datablock is deposited. Provisions can be made,if deemed necessary, to decrement reference counts in the a typical casewhere the associated version is never deposited.

[0131] Note also that the “expiration plus current count” mechanism forcontrolling the deletion of datablocks depends only on access to versioninformation at the moment when a current version is superseded. Noaccess to this information is needed at any other time, and so this isconsistent with a privacy mechanism that does not allow any access toversion information or the composition of versions except while a newobject version is actually being written.

[0132] For example, suppose that for each object, the version list whichdescribes which datablocks comprise each version is stored by thestorage servers in an encrypted form, using an encryption key whichclients give to the storage servers only at the moment that the versionis accessed. If the storage servers do not store this key, then thisinformation is unavailable at any other time, thus improving the privacyof the storage system. This brief window of visibility is, however,sufficient to allow the storage servers to assign expiration times todatablocks, and to adjust current-reference counts for datablocks. Ifthis process occurs independently within each storage clique, thennon-current “backup” versions and their associated datablocks will obeythe retention constraints imposed by preset retention policies: backupdata is safe, but unneeded datablocks are server-visibly marked as beingdeletable, and may be automatically deleted.

[0133] As another example, suppose that there is no encryption beingused, but that some datablocks are encoded using an erasure correctingcode which is very space efficient and loss resistant, but whichrequires a lot of work to recreate the original data. In this case, itis data safety rather than security that makes it difficult to accesssome version and directory information. No such access is needed,however, to allow non-current datablocks to expire and be deleted.

Automatic Metablock Deletion

[0134] The mechanism outlined in the previous section allows alldatablocks that are referenced only by expired historical versions to bedeleted and their space reclaimed. A similar mechanism can be used toallow deletion of metablocks for objects that contain only expiredversions, while protecting unexpired history. In this scheme, ametablock has a current-reference count and an expiration time, justlike a datablock. We'll call the current-reference count a “link count,”and allow the storage client to directly control this count. Theexpiration time, however, will be governed by the clique.

[0135] Whenever a reference to an object appears in a new directoryversion which did not appear in the previous version, the storage clientrequests that the clique increment the link count associated with thatobject's metablock. Similarly, when a reference disappears from acurrent directory, the storage client asks the clique to decrement thelink count. Meanwhile, whenever a new version of the object is written,or the expiration time of any version is made later, the clique adjuststhe expiration time associated with the object's metablock to be thelater of the existing expiration and the new one.

[0136] When a metablock's link count reaches zero, then (according tothe storage clients) there are no current directories that referencethis object—it has been deleted. The clique updates the expiration timeassociated with the metablock, since the current version has beensuperseded and given an expiration time. At this point the object has nocurrent version. When the expiration time associated with the metablockhas also passed, the object has no unexpired versions either. At thispoint the metablock (and its associated version list) can be deleted,and the space associated with them reclaimed.

[0137] This scheme also addresses a problem that may occur when anobject is first created with block information but no version list. Ifno version of the object is written within some reasonable period (e.g.,a month), the object can be presumed to be an orphan and deleted. Adefault expiration time for the metablock can be used to accomplishthis.

Overriding Version Retention Policy

[0138] An administrative mechanism is provided to allow an authorizedclient to delete non-current versions before they have expired, and toreclaim space freed by this action. Unauthorized clients are notpermitted to delete unexpired versions.

[0139] To support this mechanism, datablock expirations are stored in amanner that allows expirations that have been assigned to a datablock tolater be undone, with the expiration reverting to the latest expirationtime assigned which has not been unassigned. This is made easier by thefact that datablock expirations can be stored with low resolution: spacedoesn't need to be freed instantly when the last version that referencesa datablock is deleted. For example, suppose that all datablocks expireeither 10 days, 100 days or 1000 days after they are first created, ornever. This expiration information can be represented using twoadditional reference counts per datablock, to give a total of three.

[0140] In this example, the first reference count, which has beendiscussed earlier, is used to represent version-references to thedatablock which do not expire. These include references from currentversions, references from versions which are marked as “permanent,” andreferences from versions which have an expiration of greater than 1000days after the creation time of the datablock—these references are“rounded up” to permanent. The second reference count is used torepresent version-references which are between 100 and 1000 days afterthe creation time of the datablock. The third reference count representsreferences between 10 and 100 days after the creation time.

[0141] If all three counts are zero, the datablock expires 10 days afterits creation time. If the first two counts are zero but not the third,the datablock expires 100 days after its creation. If the first count iszero but not the second, the block expires after 1000 days. If the firstcount is non-zero, the datablock doesn't expire at all. When a versionis deleted, the expiration time of the version is included in a“decrement request” applied to each of its constituent datablocks. Oneof the three reference counts may be decremented, depending on thedifference between the version expiration time and the block creationtime. Similarly, if a new non-current version is written, its expirationtime is included in an “increment request” applied to each of itsconstituent datablocks. To change the expiration time of a version, itsdatablocks are first incremented using the new expiration time, and thendecremented using the old.

[0142] A sample history of expiration times for a datablock is given inFIG. 4. In the example, the datablock is first created on day 47 of2003. All three expiration counters are initially zero, and so thedatablock is scheduled to expire 10 days after its creation. In event 2,a version is added which references this datablock, and which expires onday 109 of 2003. This is rounded up to 100 days after the datablockcreation, and the third counter is incremented. A version that expireson day 140 of 2003 increments the same counter. In event 5, the versionthat was added in event 2 is deleted, and the version reference isremoved. The 100 day counter is decremented, and the datablockexpiration time is determined by the highest order non-zero count. Inevent 9, a current version is added which references this datablock. Inevent 10, this version becomes non-current and is assigned anexpiration. In event 11, a version is added which expires on day 50 of2003. None of the counters is incremented since it expires less than 10days after datablock creation. If no further version references to thisdatablock are added or deleted after event 11, then the datablockexpires on day 147 of 2003 and may be discarded and its space reusedanytime thereafter.

[0143] If two independent copies of a datablock are created and latermerged, the creation time of the later-created block is assigned to themerged block—this associates an expiration time with each count. Theexpiration times associated with the earlier-created block's counts are“rounded up” to determine which counts to add together. A simplealgorithm is to just add together corresponding counts. A subsequentdecrement may then end up being applied to a lower-order count than thecorresponding earlier increment, but this will never free a block early.

[0144] Clients may be administratively authorized to delete individualversions that expire up to some distance in the future. Deletingversions will result in the decrementing of datablock reference counts.The resulting state of the counts will indicate when each datablock willexpire, or if it has already expired. Clients may also beadministratively authorized to globally delete versions that expire upto some distance in the future. This results in the expiration time ofall datablocks in the storage system being reinterpreted, with thosethat fall within the global deletion interval being considered alreadyexpired, and available to be discarded and reused. For example, such aclient might declare all versions that expire within the next week to bealready expired. For a snapshot retention policy, this means that allsnapshots that expire within a week are instantly deleted, and theirspace reclaimed.

[0145] For increased data safety, it may be desirable to distinguishbetween versions that are not scheduled to expire because they have beenmarked to be kept permanently, and versions which are not scheduled toexpire because they are current, and so their expiration is yet to bedetermined. In the former case, normal clients should never be able toaffect the corresponding datablock reference counts, whereas in thelatter case they should. If the requirement for adequate authorizationextends all the way to the datablock level, then references for thesetwo kinds of cases can be accumulated separately. Alternatively,hash-based reference counting can be used.

Reference Counting with Hashes

[0146] Cryptographic hashing can be used to implement a form ofreference counting in which version retention policies are enforced bothat the version and at the datablock level. No matter how the datablocksare distributed across storage sites, each site containing a datablockindependently enforces the version retention policies relevant to thatblock. Cryptographic reference counting prevents malfunctioning orcompromised servers from instructing correctly operating servers toprematurely delete datablocks. This protection depends on the propertyof a cryptographic hash as a “digital fingerprint” which uniquelyidentifies a message.

[0147] In cryptographic reference counting, all information that theserver storing the datablock needs to enforce the retention policy iscommunicated with each increment or decrement request. The servercomputes a cryptographic hash of this information and this value isadded or subtracted from a reference count hash total. When the hashtotal reaches zero, the reference count is zero. Otherwise, thereference count is known only to be non-zero. The use of cryptographichashes prevents any other server from deleting information by lyingabout the relevant enforcement information: the information specifiedwith the decrement must always agree with that specified with theincrement, or the hash total will never reach zero.

[0148] For non-current versions, the relevant enforcement-informationincludes the version expiration time, the namespace_id, and sufficientinformation to uniquely locate the datablock within an object version(handle, version id, etc.). For decrement requests, the expiration timeand namespace_id are specified separately, along with a single hash ofall the other information. This allows the storage server holding thedatablock to determine which expiration count should be affected, and tocheck whether the namespace_id has been authorized to delete a versionwith this expiration. If per-object authorization is desired, the handlecan also be specified separately. For increment requests, the expirationtime along with the hash to be added to the hash-total can simply bespecified, since no checking is needed for increments.

[0149] For datablocks referenced by current versions, each incrementrequest must—in addition to information needed to locate the datablockwithin an object version—specify the retention policy and the creationtime associated with the version being created. This information will behashed and added to the hash-total. When a decrement request is sent, itmust include this same information, and this will be used to calculatethe hash needed to decrement the current-reference hash-total, and tocalculate an expiration time. If an expiration time is explicitlyincluded in the decrement-current request, the later of the calculatedand the specified times will be used to determine which datablockexpiration count to increment.

[0150] The use of cryptographic reference counts is illustrated in FIG.5. In this example we have assumed that the reference count incrementsand decrements in FIG. 4 have been replaced with cryptographic hashvalues. In event 3, for example, a message specifying a policy, anexpiration time, a namespace identifier and a location hash is sent withthe increment request, and a hash of this message is added to the100-days counter, instead of incrementing it by one. Similarly, in event9 a message is sent with the increment request which specifies that thisis a reference from a current version, when the version was created, andwhat policy should be used to assign an expiration when the versionbecomes non-current. We assume that at event 10, the current versionfrom event 9 has been current at hour 0 of a week, but not at hour 0 ofa month, and so it is assigned an expiration time of 30 days from itscreation. All of the information specified in the increment must besupplied at the decrement or the storage server holding the datablockwill not compute the same hash that was added to the no-expiry counterin event 9, and so the no-expiry counter will remain non-zero.

[0151] Note that with this scheme, it is not necessary to use separatehash-totals for current-version references and for references fromversions that are permanent.

Other Version Retention Policies

[0152] Version retention policies other than those discussed above areuseful. For example, one can keep the N most stable versions of eachobject (i.e., the ones that were current the longest), rather than theversions that happen to be current at snapshot moments. This is anexample of a non-snapshot policy that protects history, since amalicious or buggy storage client cannot affect how long historicalentity versions were current. In contrast, a policy of keeping the Nmost recent versions would not protect history, since history could beerased by simply depositing N empty versions.

[0153] Enforcing an “N most stable” policy requires the storage systemto have access to information about when each version of a stored entitywas current. Currency-intervals are recorded for all versions, and thestorage system looks at this information in applying the “N most stable”policy independently at each storage site, refusing to delete versionsprotected by the shared policy. This is illustrated in FIGS. 3 and 6.Suppose the policy is to keep the current version and up to twohistorical versions of each object, based on stability, with no snapshotpolicy in force. At the point when d4 is deposited in FIG. 3, one of d1,d2 or d3 must be deleted. Since d2 was current for the shortestinterval, it will be deleted. In FIG. 6, we see d2 being independentlyselected for deletion at each storage site, based on a record of theinterval during which each version was current.

[0154] Versions and datablocks can be automatically deleted by theobject storage system when using a most-stable policy. This fits withinthe same reference count framework used for snapshot based datablockmanagement. In this case, all versions protected by a most-stable policyhave indefinite lifetimes, just as current versions do under asnapshotting policy. Thus a datablock's “no expiration scheduled”reference count reflects how many references there are to the datablockin current or stable versions—only this count is affected by versionsprotected by a most-stable policy. When there exist N object versionsfor which the most stable N versions are retained, the addition of a newversion results in the automatic deletion of an old version. Theno-expiry reference counts of all datablocks that are part of thedeleted version are decremented. As usual, any datablock with ano-expiry reference count of zero and which has expired may have itsstorage space reused.

[0155] The overall policy on prohibition of deletions can involve theunion of several different prohibitions, based separately onexpirations, stability, and other factors; or on these factorsconsidered jointly. For example, the policy for an object could be tokeep the two most stable historical versions, in addition to anyversions required by a snapshot-based policy. In this case, up to twohistorical versions of each object have indefinite expirations. If acurrent version is superseded and it has lasted longer than one of twoexisting stable versions, then it becomes one of the two stableversions. The version it replaces is assigned an expiration time basedon the snapshotting policy, which looks at the interval during which itwas current and any expiration explicitly assigned to it. For example,in FIG. 3, suppose versions current at t1 are considered part ofsnapshot 1 and are given an expiration of a day; and in addition the twomost stable historical versions are kept. Then when d5 is deposited, d4replaces d3 as one of the two most stable versions, and d3 is given anexpiration of a day. Datablocks that are part of current or stableversions have this fact reflected in their no-expiry reference countsalone, whereas datablocks that are part of non-current snapshot versionsmay have this fact reflected in other expiration information.

Other Embodiments

[0156] One aspect of the invention that is illustrated by the preferredembodiment is that of protecting history by using a coordinated systemof deposit-time-dependent expiration policies that are independentlyenforced at separate storage locations. This idea can be applied rathergenerally. It could apply equally well to file systems that are aware ofthe nature and structure of stored data and to object storage systemsthat have little or no such access. It can be applied to systems adaptedmainly for seldom-changing archival data, or to systems designed forhigh-performance read-write access.

[0157] In preferred implementations, the storage system is made awarethat a new item is being stored that should be considered thereplacement for an existing item. It is not necessary that the storagesystem know which item is replacing the existing item, only that thereplacement is taking place and what is being replaced. This could beindicated by telling the system at deposit time that a deposited itemreplaces an existing item as a “current” version, or even by an entirelyseparate operation in which the storage system is informed that anexisting item is no longer current, and an expiration time should beassigned to it. Thus items could be, in general, created as “current”with a deposit time, and at some later time marked as “historical”, atwhich point the storage system assigns it an expiration time. Some itemscould also be deposited already marked as “historical”, with anexpiration (or an assumed interval of currency from which the storagesystem can infer an expiration) specified by the storage client.Expiration times could be assigned by storage clients to both currentand historical versions, and the effective expiration would always bethe latest of all assigned expirations. For example, a version might bemarked as “permanent” (infinite expiration) while it is still current,and this expiration would persist even after it is no longer current. Asanother example, a version might be given an expiration of a week whilecurrent, and the expiration assigned to it by policy when it becomeshistorical might be a month, which would override the shorterexpiration.

[0158] When the distributed storage system is backing up some other kindof source storage system, there are many different ways in which thesource information can be mapped into storage entities. For example,each file and directory of a file system might be mapped into aseparately named and separately accessible object, or an entire snapshotmight be mapped into a hierarchical tree of self-named datablocks whichrefer to each other using content based datanames, or some combinationof these two approaches might be taken. An entire backup record mighteven be stored as a single string of bytes, with different versionscorresponding to different backup records. Directories might containversion summary information or not. Additional history of the sourcestorage system could be added from tapes or from non-tape media. Manydifferent source storage systems could be backed up into the samedistributed storage system. If desired, encryption can be used to keepbackup data private.

[0159] Deposit time information can be determined and monitored in manyways. The important thing is to do something which makes it difficultfor an antagonist to manipulate this information. If system monitoringof deposit time accuracy for specific objects or sets of objects isinconvenient when history is initially being deposited, it could beturned off at object creation, and only turned on later. As long asmonitoring cannot easily be turned back off, protection of history isprovided.

[0160] The method used in the preferred embodiment to allow authorizedusers to override retention policy and recover storage space is quitegeneral. The same method is useful even with just a single referencecount associated with each datablock, with expiration occurring only forblocks that have a zero count, and happening a fixed time after blockcreation (or block merger). For fine control, a large number of countscould be used. The time-range associated with each of the expirationtime counters could be different for different datablocks, and couldchange with time, as long as at each transition old expiration periodsare rounded up when they are converted to new ones. The expiration timeassociated with a block that has all reference counts of zero does nothave to be fixed globally for all blocks, but could be recordedseparately for each block, or for sets of blocks, and could be changedwith time (with normally authorized clients perhaps allowed to increaseit). It could even depend on the expiration times of object versionsthat reference the block which expire during some time interval.

[0161] Retention and deletion policies that apply to object versions arepreferably inherited by the datablocks that constitute the objectversions: the rules that apply to datablocks must be consistent with therules that apply to the object versions that reference them. Forexample, the expiration time of a datablock must be no earlier than thatof any object version that includes it as a component. This basicproperty may be achieved rather generally, by having a storage serverthat holds an object version share the relevant rules—along with anyinformation needed to implement them—with the servers that hold thedatablocks the object version references. This sharing occurs when theobject version is created and new references to datablocks are added, sothat the rules governing the removal of these references are safelyestablished.

[0162] The preferred embodiment uses a technique of cryptographicreference counting to ensure that the same information is conveyed to astorage server when a reference to a datablock is removed as wasconveyed when the reference was added—thus making this informationavailable without placing the burden on the storage server to retainthis information. This takes advantage of the property of acryptographic hash as a unique fingerprint associated with a particularmessage. This technique prevents incorrect information in a decrementrequest from ever generating the same hash that was computed when thecorresponding increment was performed. This technique can clearly beused to govern any kind of distributed reference counting, whereaccidental or malicious decrements must be guarded against. A relatedtechnique, which provides less protection, is to associate a shortnon-cryptographic hash with each conventional reference count, with thehash used in the same manner as the cryptographic hashes discussed. Boththe conventional count and the associated hash must be zero for thecount to be considered to have a value of zero. This guards against manycauses of accidental deletion due to bugs in the software, but notagainst malicious attacks: short hashes don't prevent the falsificationof messages. Reference counts with hashes are also useful even whendatablocks aren't shared, since they match add-block and delete-blockrequests, and also ensure that any policy and background informationgoverning the deletion is available, or no deletion will occur.

[0163] The description of preferred embodiments did not depend on howobject versions are broken up into constituent datablocks. This can beendone in a content-independent manner (e.g., fixed length segments), orbased on content or even meta-information (such as file type). Forexample, email messages might be broken up at attachment boundaries, toincrease the number of shared datablocks. Object versions may even bebroken up at a set of byte-strings chosen pseudo-randomly, as is donefor example in the file system discussed by Athicha Muthitacharoen,Benjie Chen and David Mazieres in their paper, “A Low-Bandwidth NetworkFile System”, that appeared in 2001 in “Symposium on Operating SystemsPrinciples,” pages 174-187.

[0164] It is to be understood that the foregoing description is intendedto illustrate a few possible implementations of the invention. These anda great many other implementations are within the scope of the appendedclaims.

What is claimed is:
 1. A method by which a disk-based distributed datastorage system is organized for protecting historical records of storeddata entities, the method comprising: recording distinct states of anentity, corresponding to different moments of time, as separate entityversions coexisting within the distributed data storage system; andassigning expiration times to the entity versions independently withineach of a plurality of storage sites according to a shared set of rules,before which times deletion is prohibited.
 2. The method of claim 1 inwhich the shared set of rules requires that unexpired entity versionsnot be changed.
 3. The method of claim 1 in which the storage system isadapted for storing an unstructured-set of entities.
 4. The method ofclaim 3 in which the unstructured set comprises more than a millionentities.
 5. The method of claim 3 in which the unstructured setcomprises more than a billion entities.
 6. The method of claim 1 inwhich the storage system associates an entity with an identifier chosenby the storage client.
 7. The method of claim 1 in which the storagesystem associates an entity version with an identifier that depends on ahash of its contents.
 8. The method of claim 1 in which a client of thedistributed storage system defines mechanisms to organize the storagesystem into a hierarchical file system, with separately accessibleentities playing the roles of files and directories.
 9. The method ofclaim 1 or 8 in which expiration times of entity versions can beextended, and extension periods for different versions can be specifiedindependently.
 10. The method of claim 9 in which an expiration time isextended at the request of a client of the storage system.
 11. Themethod of claim 1 in which information about the entity is replicated toa plurality of storage sites, with the set of sites chosen based on ahash.
 12. The method of claim 1 or 8 in which entity versions can beaccessed separately, without needing to access a larger aggregate first.13. The method of claim 1 in which the plurality of storage sites arelocated in different cities.
 14. The method of claim 1 in which nosingle individual is allowed physical access to all of the plurality ofstorage sites.
 15. The method of claim 1 in which administrativemechanisms exist for overriding the deletion prohibition.
 16. The methodof claim 1 in which no single individual is given the authority tooverride the deletion prohibition at all of the plurality of storagesites.
 17. The method of claim 1 in which the versions of the entity areassigned deposit times, and the version with the latest deposit time isconsidered current.
 18. The method of claim 17 in which non-currentversions are assigned expiration times.
 19. The method of claim 17 inwhich the deposit time is specified by a client of the distributedstorage system.
 20. The method of claim 17 in which the deposit time isbased on the time the deposit reaches a storage site.
 21. The method ofclaim 19 in which the deposit time is constrained to agree with theactual time that the deposit reaches a storage site, to withinpredetermined limits.
 22. The method of claim 21 in which the actualtime is determined by clocks at the storage site, operating withoutreference to an external time standard.
 23. The method of claim 21 inwhich the actual time is determined by clocks at the storage site, witha limit to a total correction applied per fixed period using an externaltime standard.
 24. The method of claim 21 in which no constraint isimposed if the deposit time specified by the client is earlier than thelatest deposit time of any existing version of the entity.
 25. Themethod of claim 19 in which the entity is used to record the history ofa file in a source file system, and an historical version of the file isadded from a separate record of the file system's history with a deposittime that precedes the most current version of the entity.
 26. Themethod of claim 21 in which the imposition of the constraint begins at apredefined event, before which event versions of the entity aredeposited with deposit times that violate the constraint.
 27. The methodof claim 26 in which the predefined event is the deposit of a version ofthe entity with a deposit time specified that agrees with the actualtime, to within predetermined limits.
 28. The method of claim 26 inwhich the predefined event is a request from a storage client to beginmonitoring deposit times for the entity.
 29. The method of claim 25 or27 in which a client of the distributed storage system deposits recordsof a source file system's history into the storage system, with entitiescorresponding to files and directories, and the deposit times specifiedfor versions of entities correspond to times associated with therecords.
 30. The method of claim 29 in which two distinct entities, eachof which holds records of the content of a file in the source filesystem during different time intervals, are linked within a thirdentity.
 31. The method of claim 30 in which the third entity isassociated with a directory in the source file system.
 32. The method ofclaim 17 in which the expiration time assigned to a non-current versiondepends on when it was superseded as the current version.
 33. The methodof claim 32 in which the expiration time assigned to the non-currentversion depends on the deposit time that was assigned to it when it wascurrent.
 34. The method of claim 33 in which the expiration timeassigned to the non-current version depends on the deposit time assignedto the version that superseded it as the current version.
 35. The methodof claim 33 in which the expiration time assigned to the non-currentversion depends on the actual time when it was superseded as the currentversion.
 36. The method of claim 17 in which the storage client suppliesinformation that allows the storage system to associate a version withthe version that it supersedes as the current version.
 37. The method ofclaim 36 in which the information supplied by the storage client allowsthe storage system to order the versions of the entity by deposit time.38. The method of claim 34 wherein the expiration time depends on thelength of the time interval during which the version was current. 39.The method of claim 34 wherein the expiration time depends upon whichdefined snapshot moments the version was current during.
 40. The methodof claim 32 wherein the expiration time depends on the deposit times ofnon-current versions of the entity.
 41. The method of claim 1 wherein aversion is deposited, and the expiration time for it is set by thestorage client.
 42. The method of claim 1 wherein a version isdeposited, and a time interval during which it is presumed to have beencurrent is assigned by the storage client.
 43. The method of claim 42wherein the expiration time depends on the time interval during which aversion is presumed to have been current.
 44. The method of claim 1 inwhich a plurality of versions of a first entity which are depositedduring a time interval all have their expiration times extended to atleast a first expiration time.
 45. The method of claim 44 in which asecond entity which records hierarchical directory information includingthat of the first entity has a version deposited during the timeinterval which expires earlier than the first expiration time.
 46. Themethod of claim 45 in which summary information is stored in a versionof the second entity that does not expire before the first expirationtime, that is sufficient to recreate hierarchical directory informationof the version that does.
 47. The method of claim 34 in which a versionmakes reference to constituent blocks of stored content, with each blockassigned a reference count which reflects the number of references thereare to the block in any version.
 48. The method of claim 47 in which theversion is deleted by a storage client, the reference counts assigned toits constituent blocks of stored content are decremented, and a blockwith reference count of zero is discarded and its storage space isreused.
 49. The method of claim 34 in which versions make reference toconstituent blocks of stored content, with each block assigned areference count which reflects the number of references there are to theblock in current versions.
 50. The method of claim 49 in which eachblock is also assigned an expiration time that depends on the latest ofexpiration times associated with versions which make reference to it.51. The method of claim 50 in which a block which has a reference countof zero and an expiration time which has passed is discarded, and itsstorage space is reused.
 52. The method of claim 48 or 51 in which thereference counts for blocks of stored content are incremented when theblocks are deposited.
 53. The method of claim 51 in which the expirationtime for a block of stored content is set to a default non-zero valuewhen the block is deposited.
 54. The method of claim 17 in whichentities are associated with entity version records, with each entityversion record storing the association between an entity identifierfreely chosen by a storage client and the versions of the entity. 55.The method of claim 54 in which each entity version record is assigned areference count which reflects the number of references there are to thecorresponding entity from within current entity versions.
 56. The methodof claim 55 in which each entity version record is also assigned anexpiration time that depends on the latest of all of the expirationtimes associated with the versions of the entity recorded in the versionrecord.
 57. The method of claim 56 in which an entity version recordwith reference count of zero and an expiration time which has passed isdiscarded and the storage space is reused.
 58. The method of claim 57 inwhich the expiration time for an entity version record is set to adefault non-zero value when it is created.
 59. The method of claim 48 or51 in which the blocks of stored content are strings of bytes with apredetermined maximum length.
 60. The method of claim 59 in which ablock is referenced using a block name which depends upon a hash of thecontent of the block.
 61. The method of claim 60 in which the blockcontent has been encrypted using a key derived from its unencryptedcontent.
 62. A method by which a disk-based distributed data storagesystem is organized for protecting historical records of stored dataentities, the method comprising: recording distinct states of an entity,corresponding to different moments of time, as separate entity versionscoexisting within the distributed data storage system; associatingtime-intervals with entity versions, corresponding to the times duringwhich each entity version was considered current; sharing a set of rulesfor retaining entity versions among a plurality of storage sites; anddesignating some entity versions as deletable and some as undeletableindependently at each of the plurality of storage sites.
 63. The methodof claim 62 in which, except for deletion, entity versions areimmutable.
 64. The method of claim 62 in which expiration times are alsoassigned to some entity versions, independently within each of theplurality of storage sites, according to a shared set of rules, beforewhich times deletion is prohibited.
 65. The method of claim 62 in whichno single individual is given the authority to override the deletionprohibition at all of the plurality of storage sites.
 66. A method bywhich a disk-based data storage system is organized for protectinghistorical records of stored data entities, the method comprising:recording distinct states of an entity, corresponding to differentmoments of time, as separate entity versions coexisting within the datastorage system; assigning expiration times to the entity versions,before which times deletion is prohibited; and assigning expirationtimes to blocks of stored content that constitute the entity versions,with at least one block shared between different entities.
 67. Themethod of claim 66 in which the data storage system is distributed andthe expiration times are assigned independently within each of aplurality of storage sites according to a shared set of rules.
 68. Themethod of claim 66 in which the expiration time assigned to a blockreflects the latest of the expiration times associated with a pluralityof versions which make reference to it.
 69. The method of claim 66 inwhich a block is assigned a reference count which reflects the number ofreferences there are to the block in a plurality of versions which arenot scheduled to expire.
 70. The method of claim 69 in which a block isassigned a reference count which reflects the number of references thereare to the block in a plurality of versions which are scheduled toexpire during some specified finite time period.
 71. The method of claim69 in which the block is also assigned a default expiration time thatdepends on a time of origin associated with the block itself.
 72. Themethod of claim 71 in which the default expiration time depends upon theexpiration times assigned to each of a plurality of versions which makereference to the block.
 73. The method of claim 71 in which a block witha reference count of zero and a default expiration time which has passedis discarded and its storage space is reused.
 74. The method of claim 71in which an authorized storage client causes a block to be discardedwhich has a default expiration time which has not yet passed.
 75. Themethod of claim 71 in which an authorized storage client causes aversion to be deleted for which the assigned expiration time has not yetpassed.
 76. The method of claim 75 in which a block referenced by thedeleted version is discarded and its storage space is reused.
 77. Amethod for keeping track of when all references of a specified categorymade to elements have been removed, the method being designed to fail ina manner that does not falsely conclude there are no references, themethod comprising: computing a hash value that identifies the source ofa reference; combining hash values using a first operation to record theaddition of references; combining hash values using a second operationto record the removal of references; and concluding that referenceadditions for an element have been matched by reference removals. 78.The method of claim 77 in which the hash is a cryptographic hash. 79.The method of claim 77 in which the first operation includes countingthe number of reference additions.
 80. The method of claim 77 in whichthe first operation includes adding together hashes.
 81. The method ofclaim 77 in which the first operation includes adding corresponding bitsof hashes together modulo
 2. 82. The method of claim 77 in which thehash value uniquely identifies the reference source.
 83. The method ofclaim 77 in which additional information not needed to identify thereference source is included in the identifying hash.
 84. The method ofclaim 83 in which hash values are combined at a physical location thatis separated from a source of references.
 85. The method of claim 84 inwhich the additional information is examined at the location where thehash values are combined, and a decision is made to not combine a hashvalue.
 86. The method of claim 84 in which the additional information isexamined at the location where the hash values are combined, anddetermines which categories of combined hash will be affected.
 87. Themethod of claim 86 in which a reference-removal operation is performedon one category of combined hash and a reference-addition operation isperformed on another.
 88. The method of claim 84 in which referencesources and combined hashes are distributed among a collection ofcomputers.
 89. The method of claim 88 in which the computers are serversin a disk-based data storage system.
 90. The method of claim 89 in whichthe data storage system is organized for protecting historical recordsof stored data entities.
 91. The method of claim 89 in which distinctstates of an entity are recorded, corresponding to different moments oftime, as separate entity versions coexisting within the data storagesystem.
 92. The method of claim 91 in which expiration times areassigned to the entity versions, before which times deletion isprohibited.
 93. The method of claim 92 in which expiration times areassigned according to a shared set of rules.
 94. The method of claim 93in which expiration times are assigned to blocks of stored content thatconstitute the entity versions.
 95. The method of claim 94 in which ahash value identifies the reference of an entity version to a block thatis shared with other entities.
 96. The method of claim 95 in whichinformation about the shared set of rules is included in thereference-identifying hash.
 97. The method of claim 95 in whichinformation that allows the general deletion prohibition to be ignoredis included in the reference-identifying hash.
 98. The method of claim95 in which reference additions to the shared block have been matched byreference removals, and the shared block is discarded and its storagespace is reused.
 99. A method by which more than one client programconnected to a network stores the same data item on a storage device ofa data repository connected to the network, the method comprising:encrypting the data item using a key derived from the content of thedata item; determining a digital fingerprint of the data item; storingthe data item on the storage device at a location or locationsassociated with the digital fingerprint; and assigning an expirationtime to the data item, before which time deletion is prohibited. 100.The method of claim 99 in which rules governing expiration and deletionare distributed among a plurality of storage sites.
 101. The method ofclaim 99 in which the expiration time assigned to the data item dependsupon expiration times assigned by the client programs.
 102. A method bywhich more than one client program connected to a network stores thesame data item on a storage device of a data repository connected to thenetwork, the method comprising: determining a digital fingerprint of thedata item; testing for whether the data item is already stored in therepository by comparing the digital fingerprint of the data item to thedigital fingerprints of data items already in storage in the repository;challenging a client that is attempting to deposit a data item alreadystored in the repository, to ascertain that the client has the full dataitem; and assigning an expiration time to the data item, before whichtime deletion is prohibited.
 103. The method of claim 102 in which rulesgoverning expiration and deletion are distributed among a plurality ofstorage sites.
 104. The method of claim 102 in which the expiration timeassigned to the data item depends upon expiration times assigned by theclient programs.
 105. A method by which more than one client programconnected to a network stores the same data item on a storage device ofa data repository connected to the network, the method comprising:determining a digital fingerprint of the data item; storing the dataitem on the storage device at a location or locations associated withthe digital fingerprint; associating the data item with each of aplurality of access-authorization credentials, each of which is uniquelyassociated with an access owner; assigning an expiration time to thedata item, before which time deletion is prohibited; and preparing adigital time stamp of a plurality of records associating data-items andcredentials, to allow a property of these records to be proven at alater date.
 106. The method of claim 105 in which rules governingexpiration and deletion are distributed among a plurality of storagesites.
 107. The method of claim 105 in which the expiration timeassigned to the data item depends upon expiration times assigned by theclient programs.
 108. A method by which more than one client connectedto a network stores the same data item on a storage device of a datarepository connected to the network, the method comprising: determininga digital fingerprint of the data item; testing for whether a data itemis already stored in the repository by comparing the digital fingerprintof the data item to the digital fingerprints of data items already instorage in the repository; associating with a data item an informationaltag which may be read by at least some client programs; and assigning anexpiration time to the tagged data item, before which time deletion isprohibited.
 109. The method of claim 108 in which rules governingexpiration and deletion are distributed among a plurality of storagesites.
 110. The method of claim 108 in which the expiration timeassigned to the tagged data item depends upon expiration times assignedby the client programs.
 111. A method by which a client connected to adata repository over a lower speed network connection may provide higherspeed access to a data item for application processing than is possibleover the relatively low speed connection to the network, the methodcomprising: determining a digital fingerprint of the data item; testingfor whether the data item is already stored in a repository by comparingthe digital fingerprint of the data item to digital fingerprints of dataitems already in the repository; only if the data item is not already inthe repository, transferring the data item over the lower speedconnection from the client to the repository; assigning an expirationtime to the data item, before which time deletion is prohibited; makinga higher speed connection between an application server and the datarepository; executing an application on the application server toprocess the data item stored on the data repository; and returning atleast some of the processed data to the client across the lower speedconnection.
 112. The method of claim 111 in which rules governingexpiration and deletion are distributed among a plurality of storagesites.
 113. The method of claim 111 in which the expiration timeassigned to the data item depends upon expiration times assigned by theclient programs.
 114. A method by which multiple clients browse contenton a network such as the Internet, the method comprising: each of themultiple clients accessing content on the network via one or more proxyservers; determining the digital fingerprint of an item of contentpassing through the proxy server; storing the item of content in acontent repository connected to the proxy server at a locationassociated with the digital fingerprint; assigning an expiration time tothe item of content, before which time deletion is prohibited; testingfor whether a content data item is already stored in the repository bycomparing the digital fingerprint of the content data item to thedigital fingerprints of content data items already in storage in therepository; and associating a content data item already stored in therepository with an access authorization credential uniquely associatedwith an access owner.
 115. The method of claim 114 in which rulesgoverning expiration and deletion are distributed among a plurality ofstorage sites.
 116. The method of claim 114 in which the expiration timeassigned to the item of content depends upon expiration times assignedby the multiple clients.
 117. A method by which clients store contentitems which are broken into up into smaller data items in a datarepository connected to the network, the method comprising: determininga digital fingerprint of a data item; testing for whether a data item isalready stored in the repository by comparing the digital fingerprint ofthe data item to the digital fingerprints of data items already instorage in the repository; and assigning an expiration time to a dataitem, before which time deletion is prohibited.
 118. The method of claim117 in which rules governing expiration and deletion are distributedamong a plurality of storage sites.
 119. The method of claim 117 inwhich the expiration time assigned to the data item depends uponexpiration times assigned by the multiple clients.
 120. The method ofclaim 117 in which the expiration times assigned to data items thatcomprise a content item depend upon an expiration time assigned to thecontent item.
 121. The method of claim 117 in which the content item isbroken up in a manner that is independent of the content.
 122. Themethod of claim 117 in which the content item is broken up in a mannerthat depends on the content type.
 123. The method of claim 117 in whichthe content item is broken up at boundaries defined by predeterminedbyte strings.
 124. The method of claim 123 in which the choice of whichbyte strings constitute boundaries depends upon the value of a hashfunction acting on the byte strings.
 125. A method for ensuring thatrules that prevent premature deletion of entity versions are enforced bycorrectly operating servers that store the blocks of content thatcomprise the entity versions, the method comprising: computing a hashvalue that identifies the source of a reference to a block of content;incorporating into the hash value a description of rules or parametersthat are needed in order to enforce rules; and communicating informationwhich allows the hash value to be computed, to a server that stores theblock of content.
 126. The method of claim 125 in which the hash is acryptographic hash.
 127. The method of claim 125 in which the hash valueuniquely identifies the reference source.
 128. The method of claim 125in which a block of content is identified by a digital fingerprint thatinvolves a hash of its content.
 129. The method of claim 125 in which ablock of content is assigned an expiration time, before which timedeletion is prohibited.
 130. The method of claim 125 in which the blocksof content are distributed among a plurality of storage sites.
 131. Themethod of claim 125 in which an expiration time assigned to an entityversion is also assigned to each of its constituent blocks of content.132. The method of claim 125 in which the information which allows thehash value to be computed is included in a request to delete the blockof stored content.
 133. The method of claim 132 in which a serverstoring the block of content denies a request that violates a rule orparameter specified in the information supplied when the block wascreated.
 134. The method of claim 125 in which distinct states of anentity are recorded, corresponding to different moments of time, asseparate entity versions coexisting within a data storage system. 135.The method of claim 134 in which the rules governing deletion of anentity version depend upon when the entity version was created.
 136. Themethod of claim 125 in which hash values that identify references toblocks of stored content are combined as part of a reference countingscheme.
 137. The method of claim 136 in which some reference counts areassociated with expiration times, and their values are ignored aftersome point in time.
 138. The method of claim 1 or 125 in which theconnection between an entity version and a constituent block of contentis not visible to a server storing the block of content.
 139. The methodof claim 138 in which the stored block of content expires and the serverstoring it discards it and reuses its storage space.
 140. The method ofclaim 36 in which the information supplied by the storage client thatassociates a version with a superseded version is discarded while thetwo versions are retained.
 141. A method by which a distributeddisk-based data storage system is organized for protecting historicalrecords of stored data entities, the method comprising: recordingdistinct states of an entity, corresponding to different moments oftime, as separate entity versions coexisting within the data storagesystem; assigning expiration times to the entity versions, before whichtimes deletion is prohibited; assigning expiration times to blocks ofstored content that constitute the entity versions; and assigning areference count to a block of stored content that reflects the number ofreferences there are to the block in entity versions which are scheduledto expire during some specified finite time period.
 142. The method ofclaim 141 in which the block is also assigned a reference count thatreflects the number of references there are to the block which are notscheduled to expire.
 143. The method of claim 141 in which the block isalso assigned a default expiration time which sets an earliest time thatthe block can expire, even if all expiration related reference countsare zero.
 144. The method of claim 141 in which the data storage systemis distributed and the expiration times are assigned independentlywithin each of a plurality of storage sites according to a shared set ofrules.
 145. The method of claim 141 in which an authorized storageclient causes a block to be discarded and its space reused when itsexpiration time has not yet passed.
 146. The method of claim 141 inwhich an authorized storage client overrides the deletion prohibitionand causes an entity version to be deleted when its expiration time hasnot yet passed.
 147. The method of claim 146 in which a block of storedcontent referenced by the deleted version is discarded and its storagespace is reused.
 148. A method by which a disk-based data storage systemis organized for protecting historical records of stored data entities,the method comprising: recording distinct states of an entity,corresponding to different moments of time, as separate entity versionscoexisting within the data storage system; and assigning finiteexpiration times to entity versions based on information supplied by thestorage client, before which times deletion is prohibited and afterwhich times deletion is allowed.
 149. The method of claim 148 in which aversion is deposited, and the expiration time for it is set by thestorage client.
 150. The method of claim 148 in which a version isdeposited, and a time interval during which it is presumed to have beencurrent is assigned by the storage client.
 151. The method of claim 150in which the expiration time is assigned by a storage server and dependson the time interval during which a version is presumed to have beencurrent.
 152. The method of claim 150 in which the entity is used torecord the history of a file in a source file system, and an historicalversion of the file is added from a separate record of the file system'shistory.
 153. The method of claim 152 in which the added historicalversion has an interval during which it is presumed to be currentspecified that predates that of an existing version of the entity. 154.The method of claim 148 in which expiration times of entity versions canbe extended, and extension periods for different versions can bespecified independently.
 155. The method of claim 148 in which unexpiredentity versions cannot be changed.
 156. The method of claim 148 in whichthe storage system is adapted for storing an unstructured-set ofentities
 157. The method of claim 148 in which the connection between anentity version and a constituent block of content is not visible to aserver storing the block of content.
 158. The method of claim 148 inwhich a plurality of versions of a first entity which are depositedduring a time interval all have their expiration times extended to atleast a first expiration time.
 159. The method of claim 158 in which asecond entity which records hierarchical directory information includingthat of the first entity has a version deposited during the timeinterval which expires earlier than the first expiration time.
 160. Themethod of claim 159 in which summary information is stored in a versionof the second entity that does not expire before the first expirationtime, that is sufficient to recreate hierarchical directory informationof the version that does.
 161. The method of claim 148 in which versionsmake reference to constituent blocks of stored content, with each blockassigned a reference count.
 162. The method of claim 161 in which eachblock is also assigned an expiration time that depends on the latest ofexpiration times associated with versions which make reference to it.163. The method of claim 162 in which a block which has a referencecount of zero and an expiration time which has passed is discarded, andits storage space is reused.